Rendering engine component abstraction system

ABSTRACT

The subject matter discloses a system for testing an application or a game build on top of a rendering engine, the system comprises an agent configured for running on a first computing device; the first computing device is configured for running the game or said application; a test framework installable on a second computing device, the test framework is configured for receiving one or more instructions related to a test of said game or the application from a user and for transmitting the one or more instructions to the agent; wherein the agent is further configured for communicating with the test framework for extracting one or more object properties of the game or the application in response to the one or more instructions and for providing to the test framework to access to the one or more object properties.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 16/030,873, filed on Jul. 10, 2018, entitled“RENDERING ENGINE COMPONENT ABSTRACTION SYSTEM,” which claimed thebenefit of U.S. Provisional Patent Application No. 62/537,019, filed onJul. 26, 2017, entitled “RENDERING ENGINE COMPONENT ABSTRACTION SYSTEM,”each of which is hereby incorporated by reference.

FIELD OF THE INVENTION

The present disclosure relates to rendering engines in general, and totesting the products built on top of the rendering engines, inparticular.

BACKGROUND OF THE INVENTION

Rendering or image synthesis is the automatic process of generating aphotorealistic or non-photorealistic image from a 2D or 3D model (ormodels in what collectively could be called a scene file) by means ofcomputer programs.

Rendering engines provide all the functionality that is required fordesigning a video game or an application isolating the developer fromlow level mathematical, physics and graphics implementation need. Thefunctionality includes the rendering, the sound and the graphicstogether with user interaction, physics models and mathematicalcalculations.

SUMMARY OF THE INVENTION

The term computing device refers herein to a device that includes aprocessing unit. Examples for such device are a personal computer, alaptop, a server, a tablet and a cellular device and IOT (internet ofthings) device.

The term RECAS refers herein to Rendering Engine Component AbstractionSystem.

The term external system refers herein to a system that utilizes theRECAS for testing.

One exemplary embodiment of the disclosed subject matter is a method andsystem for automating the testing of products that are built on top andthat are generated by a rendering engine. The products are video gamesor any graphical application. The testing of the products may includethe aspects of functionality, localization, stability and performance.

The term object property refers herein to an element in a game or anapplication that can be referred to and manipulated by the operatingsystem typically as a result of scenario in a game or an applicationcaused by an interaction of the user. Examples of such object propertiesare visibility on a screen/scene, size, position (x/y coordinates or x,y, z coordinates in 3d environment), clickability and touchability.

Test automation is one of the critical elements of software developmentprocess dictated by customer need of assuring high quality of theproduct within short period of time. Manual based testing contradictsthis expectation so in many domains of software, test-automation must beintroduced. This includes auto-testing of native applications installedon different platforms/operating systems and auto-testing of web basedapplications executed in web browser environment.

Native application and Web base applications are well understood andthere are techniques allowing efficient automation of testing. As anexample—native apps available on Android or IOS platforms are connecteddirectly with window management/UI controller layer, exposing interfaceto its UI environment and its properties and functions, allowingsimulation of real app usage behavior. The same happens in all appsavailable via web browser. Web browser being execution environment forthe application gives access to its components using standard DOM(Data-Object-Model) like structures and allows discovery of topology ofthe UI elements, their activity parameters and features.

One technical problem disclosed by the present invention is how toprovide access to the graphical elements for auto-testing an applicationor a game that is developed on a rendering engine.

In case of applications (including games) built on top of renderingengines there is no direct connection to window manager/UI controllerlayer of operating system/platform. This results with disability ofelastic steering of the app behavior from external system, having noknowledge of the components visible on the screen during appinteraction.

In current industry state, there is no automation testing solutionsufficient for applications built on the top of rendering engines (eg.Unity/Unreal Engine). Known in the art automating testing solution omitthe OS (operating system) level services and use direct screen accesswhich does not allow getting any information about UI layer. Such knownin the art solutions do not provide feedback from application aboutcurrent content displayed on the screen. Existing, primitive automationtools can only perform action based on the screen coordinates withoutany validation, leading to auto-test scenario errors and the inabilityto perform some actions which are important for the testing.

Test developers may try to get some basic information about state of theapplication via system logs and parsing text data or try to usecomplicated image recognition algorithms to take screenshot and comparethe screenshot with predefined assets and make decision based on them.

Such known in the art methods are not easy to implement and not assurefull reliability during test execution. Such known in the art methodsare hard to maintain.

One technical solution, is allowing to receive topology information(such as structures that include information about the graphicalcomponent) and to interact with the application/game dynamically. Thedynamic interaction is performed by sending steering information andinputs simulation directly to window manager/UI controller layer orbypassing the window manager and UI controller layer by transferring therequests directly to RECAS subcomponent.

By using such a solution the external system (the system which is usingthe RECAS technology) may interact with the tested applications/gamesand may provide information about the tested application/games.

The solution may be implemented on any type of operating systems and/orplatforms including mobile operating system, Android, or PC or gameconsole platforms.

By using such a solution the external system allows the user to writecustomized automated test scenario tailored to user needs withoutenforcing any automatic pass/fail criteria for such test domain; thus auser can freely develop its own scenario. The behavior and logic ofexecuted scenario may be changed without the necessity to change thescenario itself.

By using such a solution the external system checks the stability of theapplication during multiple executions of the same test scenario to makesure that test is finished with success over many retries withoutapplication crash.

By using such a solution the external system provides measurements ofbasic performance metrics important from mobile applicationsperspective.

By using such a solution the external system provides the monitoring ofthe following:

-   -   FPS—Number of Frames Per Second    -   CPU—% usage of the device processors    -   Memory—% usage of the device memory Metrics list can be        enhanced:    -   Disk I/O—read/write activity on disk    -   Network I/O—up/down transfer on network    -   Battery—consumption of battery power    -   Temperature—Battery—temperature of battery    -   Temperature—CPU—temperature of CPU

By using such a solution the external system provides the checking ofcorrectness of the texts translations for different languages fordifferent device resolutions. The external system provides localizationtests based on the screenshots which are executed at given timeaccordingly to tester expectations. Screenshots are post processed withusing of the OCR module allowing detecting fonts and identifyingdiscrepancies in texts.

By using such a solution the external system verifies and validates thecorrectness of analytics events triggered in game. For example thesystem may check the format of the events and the number of parametersthat are sent in expected moment.

By using such a solution the external system provides monkey tests.Monkey tests do not require the writing of any automated test scenariofor tested application. Monkey test allow the starting of an applicationand the clicking on the randomly X/Y coordinates with defined timeinterval between steps. Monkey tests require minimal level ofintegration of Micro-service in the tested application.

By using such a solution the external system provides automaticdetection of the test execution status and the analysis of the scenariocorrectness by handing over the decision to the system

By using such a solution the external system provides interaction(downloading information about the game status) and dynamic adaptationof tests to the data displayed on the screen

By using such a solution the external system provides annotations. Theannotation is a mechanism which allows the marking of special pointsduring test executions with additional text message. There are threelevel of annotations:

-   -   Actions—sent manually from the test scenario, defined by the        test developer during writing of test scenario.    -   Screenshots—added automatically every time when test scenario        triggered screen capturing.    -   Sections—added automatically every time when test scenario        detects that screen name has been changed (as result of the        getScreen request).        By using such a solution the external system provides detailed        reports for each test execution. The reports may contain the        following information:    -   General information about launched scenario, used device, time        interval, test status with detailed information about failure        reason, and platform    -   Screenshots which are the basis for assessing the correctness of        the displayed content for, among others, localization tests or        verification of graphic-display correctness by the tester,    -   Performance metrics statistics in the summarized form (median,        average) is the basis for reports comparing resource consumption        for the same scenarios in different types of devices, or        different versions of build for a given application.    -   Performance interactive graph showing consumption of the system        resources by the tested application over time and annotations    -   A crash log file which is a grabbing module that ensures        coupling with the operating system and detection of crashes/core        dump after restarting the application. Log files according to        level of logging.    -   Last step detection which returns the last screen name achieved        during test execution, useful to find potentially weak points of        tested application.

One other technical problem is how to provide to the tester control overthe objects that are tested.

One other technical solution is allowing a tester to select the objectproperties to be exposed, According to some embodiments the tester mayselect which object properties of the game or application to test in acertain scenario.

According to some embodiments the system provides the tester an optionto select the object properties that are tested by the system. After thetester selects the object properties the system extracts the objectproperty during the playing of the application/game. The extracting maybe performed by interacting with the application/game via API(application interface commands). The system then interacts with theapplication components using extracted properties.

The system then provides the tester the option to use and/or manipulatethe object properties in order to test the game/items. The tester can,for example, activate commands on the object properties for monitoringbehavior of the object properties. The tester can also move the objectfrom one position to the other.

The system provides the tester GUI which enables him to perform thetest. The operations of the tester are translated into API commandswhich are imposed on the game/application.

One embodiment of the disclosed subject matter is a system for testingan application or a game build on top of a rendering engine, the systemcomprises an agent configured for running on a first computing device;the first computing device is configured for running the game or saidapplication; a test framework installable on a second computing device,the test framework is configured for receiving one or more instructionsrelated to a test of said game or the application from a user and fortransmitting the one or more instructions to the agent; wherein theagent is further configured for communicating with the test frameworkfor extracting one or more object properties of the game or theapplication in response to the one or more instructions and forproviding to the test framework to access to the one or more objectproperties.

According to some embodiments the test framework interacts with theobject framework for simulating controlling action. The interaction isperformed by, for example, executing UI controlling actions like swipe,multi-touch, tap or directly by the UI controller part of the operatingsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosed subject matter will be understood and appreciatedmore fully from the following detailed description taken in conjunctionwith the drawings in which corresponding or like numerals or charactersindicate corresponding or like components. Unless indicated otherwise,the drawings provide exemplary embodiments or aspects of the disclosureand do not limit the scope of the disclosure. In the drawings:

FIG. 1 shows a block diagram of an environment of the rendering enginecomponent abstraction system; in accordance with some exemplaryembodiments of the subject matter;

FIG. 2 shows a block diagram of the architecture of the rendering enginecomponent abstraction system in accordance with some exemplaryembodiments of the subject matter;

FIG. 3 shows a block diagram of the main modules of the external systemusing the rendering engine component abstraction system; in accordancewith some exemplary embodiments of the subject matter;

FIGS. 4a, 4b, 4c, 4d, 4e, 4f, 4g, 4h, 4i, 4j and 4k show a block diagramof exemplary functions of a system for rendering engine componentabstraction; in accordance with some exemplary embodiments of thesubject matter; and

FIG. 5 shows a scenario of using the external system which utilizesRendering Engine Component Abstraction System; in accordance with someexemplary embodiments of the subject matter.

DETAILED DESCRIPTION

FIG. 1 shows a block diagram of an environment for interfacing withgraphical elements of the rendering engine in accordance with someexemplary embodiments of the subject matter. The environment 100includes test framework 101 interacting with window manager UIcontroller 102 and with the Rendering Engine Component AbstractionSystem (RECAS) 103. The rendering engine component abstraction layer 103interfaces between the test frameworks 101, the window manager UIcontroller 102 and the application or game 104.

-   -   Test Framework 101 receives information about application/game        screen/scene components/elements connecting directly to the        agent, The Test Framework 101 interacts with the extracted        components/elements properties by either the agent or directly        via the window management UI controller 102 of the operating        system.

The rendering engine component abstraction layer 103 is explained ingreater details in FIG. 2 shows a block diagram of the architecture ofthe Rendering Engine Component Abstraction System; in accordance withsome exemplary embodiments of the subject matter. The system 103includes an agent 207 and a micro-service 203.

The agent 207 runs on the platform as an individual background process.In another embodiment the agent 207 is embedded into the applicationunder test (depending on the developer needs or convenience). The roleof the agent is to establish communication between external system andan instance of device on which the application or the test is tested.

The agent 207 is configured to launch/terminate tested application, toexecute controlling commands sent from external system to theapplication, to convey response data back to external system, executingscreenshots, and to measure crucial parameters from testing perspective.

The agent 207 contacts the micro-service 203. The micro service engine203 is installed in the rendering engine 202 of each device that testsan application or a game that is developed with a rendering engine. Themicro-service 203 may contact with the window manager UI controller 204of the device with the operating system 205 of the device and with thehardware 206 of the device.

The role of the micro service 203 is to interact with tested applicationor game 201 based on the received command from agent 207. The microservice engine 203 executes API interface/methods for communicatingbetween the test framework and the rendering engine. Examples of suchmethods are provided in FIG. 4

It should be noted that agent 207 may be separate process executed onthe device, interacting with application under test, or may be embeddedinto the application under test.

FIG. 3 shows a block diagram of the main modules of the external system;in accordance with some exemplary embodiments of the subject matter.

The external system for rendering engine component abstraction includesweb client 301, device management module 302, device platformapplication state detection module 305, test management module 306, testexecution module 309, test state correlation module 310, test resultdecision module 311, test report aggregation module 308, test reportgeneration module 308, rat 304 and dispatcher 303.

The Web Client module 301 is configured for providing the GUI (graphicaluser interface) for the web server. The Web Client module may besupported by web server, relational database such as Apache Server withPHP server-side scripting language.

The Web server is configured for providing a primary access point forcontractors allowing business care, system and test management, reportsvisualization and security access. Varied functionality is provided by,for example, web server MySQL database and access to other Back Endservices (modules). End user interface is professional admin dashboardwith fully responsible control panels. Responsive HTML template is basedon the CSS framework such as Bootstrap 3. It utilizes all of theBootstrap components in its design and re-styles many commonly usedplugins to create a consistent design

The Device Management Module 302 is configured for providingadministrative layer for handling of the physical devices. The DeviceManagement Module 302 is responsible for adding new physical devicesinto GTF system and for configure them.

The Device, Platform, Application State Detection Module 305 isconfigured for allowing the user to select and choose appropriate set ofdevices which are used during automated testing. The module provides theoption to define device type and model and OS version and shows currentstatus of the devices (online, offline, busy, etc.). The module is alsoresponsible for installing application on the mobile device before testexecution and make sure that process finished successfully.

The Test Management Module 306 is configured for maintaining user'sautomated scenarios in the GTF. Each tested application has own storagespace where user can upload own automated scenarios via GUI. The GTFvalidate structure of the script, check for potential errors and exposestests for test execution module.

The Test Execution Module 309 is configured for starting of theexecution of test scenario starting from launching the Agent and testedapplication, trough proceeding accordingly to the actions designed viatest developer in the automated scenario.

The Appl-Test State Correlation Module 310 is configured for conductingongoing correlation of detected states, gaming behavior and applicationlogs with the state of the testing process in the aspect of time anddetected events. The game status is critical information for theanalyzer. The Appl-Test State Correlation Module 310 evaluates andcompares the effect of stimulation (testing stage) in the game with theexpected response (testing stage), which is saved in the test scenario.

Test Result Decision Module 311 is configured for confirming or denyingthe operation of the test in a given test scenario. The decision of themodule is based on the information from the correlator of the test, gamestatus and the chosen test domain and other criteria.

Test Report Aggregation Module 308 is configured for processing resultsdata and aggregates them before attaching to report.

Test report Generation Module 307 is configured for generating reportson testing domains, builds, equipment and the like.

The RAT 304 is the Proxy server between GTF (Game Testing Framework)system and RECAS components. The RAT 304 is configured for establishingand maintaining connections with all RECAS instances, translating REST(Representation State Transfer) request into appropriate protobufcommands and hosting separate buffers (HUI—Hardware Usage Indicators(Performance metrics), LOG—Logs, AC—Analytics Collector) for eachdevice.

The Dispatcher 303 is configured for providing simultaneously testsexecution on group of devices. The dispatcher verifies separationbetween each test execution. The dispatcher isolates traffic dedicatedto different RAT servers and handles different device connections.

FIGS. 4a, 4b, 4c, 4d, 4e, 4f, 4g, 4h, 4i, 4j and 4k show a block diagramof exemplary functions of a system for rendering engine componentabstraction; in accordance with some exemplary embodiments of thesubject matter.

FIG. 4a shows a block diagram of the recas: setuuid function. Thisfunction sets unique identifier of the device, on which the applicationis installed and tested. The global uniqueness may be achieved viagenerating SHA-1, SHA-256 or other alphanumeric string with near-zeroconflict probability. According to some embodiment RAT sends the setuuidcommand, for the first time the device is used/and application isinstalled.

FIG. 4b shows a block diagram of the RECAS::getScreen function. Thisfunction provides information about menu page, visible on theapplication under test. The main data includes screen ID or name, itsphysical and virtual size in pixels, JSON/DOM like structures are sentin the response of the request, building the topology of the screenunder test. The function extracts the selected object properties fromthe scene and may also extract properties associated with the extractedobject properties. It is up to developer integrating RECAS into theapplication what kind of objects/elements/components are extracted ornot. In other words, as an example, the developer can decide to exposeall buttons available on the scene and not extract all text boxes.

FIG. 4c shows a block diagram of the RECAS::getButtonList function. Thefunction returns information about all visible buttons on the screen.The function includes data such as: button id or name, x/y coordinates,normalized center x/y coordinates, x/y size of the button. Thenormalized data provides the ability to identify button positionregardless of the actual screen resolution of the device that runs theapplication under test. Structures such as JSON/DOM are sent in theresponse of the request, building the topology of the screen under test.It is up to developer integrating RECAS if he/she wishes to extractbutton information or not.

FIG. 4d shows a block diagram of the function RECAS::getTextBoxList. Thefunction returns information about all visible text areas on the screen.The function includes data such as: text area id or name, x/ycoordinates, normalized center x/y coordinates, x/y size of the textarea. The normalized data provides the ability to identify text areaposition regardless of the actual screen resolution of the device theapplication under test is running. Structures such as JSON/DOM areincluded in the response of the request, building the topology of thescreen under test. It is up to developer integrating RECAS if he/shewishes to extract TextBox information or not.

FIG. 4e shows a block diagram of the function RECAS::simulateAction. Thefunction simulates manual user interface actions on a given device.Depending on device type/platform, these actions can mimic specializedcontrollers used in interaction with app, or mimic touch screen actions.The simulated actions depend on device types, platforms. Such actionsmay include but not limited to tap, combo-type, multi-touch, swipe,controller-left, controller-right, controller-up and controller-down.The simulation is performed by interacting with the operating system. INone example the system extracts the position of the object property anduses the position for simulating a click command. It is up to developerintegrating RECAS if he/she wishes to use RECAS to be an actionsimulator, or if he/she wishes to redirect such action simulationcommands directly to UI window manager of the operating system.

FIG. 4f shows a block diagram of the function RECAS::getUserProfile. Thefunction provides information about user properties or features. in caseof game industry applications this function returns the player profilecustom data. Such data may be but is not limited to status, coins,level, cards, etc. The data may be changed for each application or game.It is up to developer implementing RECAS, what exactly properties willbe exposed. FIG. 4g shows a block diagram of the function

RECAS::getMeasurement. The function provides measurements done againstthe application under test, assumes measurement data buffering (metricsgathering) on the external system side, set of measurements, dependingon device types/platforms: CPU consumption, memory consumption, batteryconsumption, temperature of CPU, temperature of battery, network trafficin/out, disk in/out. It is up to developer implementing RECAS, whatexactly measurements will be exposed

FIG. 4h shows a block diagram of the function RECAS::takeScreenshot.This function returns the dump of the graphic frame directly from enginerendered environment. The function allows very fast, non performanceimpacting screenshotting, bypassing system level bitmap dumpingfunctions. It is up to developer implementing RECAS, if he/she allows todo screenshots by the RECAS or not.

FIG. 4i shows a block diagram of the function RECAS::execCommand. Thefunction has transportation role allowing sending to the applicationunder test any kind of commands needed from external system perspective.The functions includes functionality for passing the applicationspecific commands, especially debugging or application control related.Examples of usage include but not limited to: goto background, gotoforeground, debug level settings, reloading, disable/enable modules,disable/enable pop-ups, change language etc. It is up to developerimplementing RECAS, what exactly commands he/she wants to handle.

FIG. 4j shows a block diagram of the functions recas:: startlog,stoplog, checklog, getlog. These functions handle log tracinginformation from tested application. The functions provide differentlevels of debug log details from application under test, engine andplatform layer. It is up to developer implementing RECAS, what exactlylogging level will be supported.

FIG. 4k shows a block diagram of the functions RECAS:: startAC, stopAC,checkAC, getAC. The functions provide analytics which is importantelement of application business, especially visible in mobileapplications and web applications. The function provides correctness ofthe analytics functionality in critical and manual testing of it is timeconsuming. The function provides automatization of analytics testing, byproviding analytics data gathering, probing and checking correctness. Itis up to developer implementing RECAS, what exactly analytics data willbe extracted.

FIG. 5 shows a scenario of using the system for rendering enginecomponent abstraction; in accordance with some exemplary embodiments ofthe subject matter. The user may also select theobject/element/component property of the game/application that have tobe tested.

At block 501 the user logs into system. The log in is afterregistration. The user selects the mobile device from which the test isexecuted, the game or application to be tested, the device on which thetested game or application runs and the test script (scenario). The useruploads the game build.

At block 505 the front end sends the request of the user to the backend. The request includes information about the test.

At block 510 the back end starts the test activity.

At block 515 the back end allocates the requested device.

At block 520 the back end installs the game build.

At block 525 the back end launches the agent on the device and startsthe execution of the test scripts that simulate the behavior of thegame.

At block 530 the communication between the RAT server and the agent isestablish. In some embodiments, from this point onward all of thecommunication between test on backend and RECAS is through RAT.

At block 535 Backend asks RECAS to launch the tested application on thedevice. The agent starts the game or application.

At block 540 the game or the application is executed according to thescript by sending commands to RECAS to control and test the application.

At block 545 which occurs when the script ends, a report is generatedand is sent to the user via the front end.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

It should be noted that, in some alternative implementations, thefunctions noted in the block of a figure may occur out of the ordernoted in the figures. For example, two blocks shown in succession may,in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved.

1. A system for testing an application or a game build on top of arendering engine, the system comprises: an agent configured for runningon a first computing device; said first computing device is configuredfor running said game or said application; a test framework installableon a second computing device, said test framework is configured forreceiving one or more instructions related to a test of said game orsaid application from a user and for transmitting said one or moreinstructions to said agent; wherein said agent is further configured forcommunicating with said test framework for extracting one or more objectproperties of said game or said application in response to said one ormore instructions and for providing to said test framework an access tosaid one or more object properties.